JOURNAL OF LEARNING ANALYTICS 

(2015). Discourse-Centric Learning Analytics: Mapping the Terrain. Journal of Learning Analytics, 2(1), 185-209. 


SSLAR 


Discourse-Centric Learning Analytics: Mapping the Terrain 

Simon Knight 

Knowledge Media Institute, Open University, United Kingdom 
sjgknight(5)gmail.com 

Karen Littleton 

Centre for Research in Education and Educational Technology, Open University, United Kingdom 

ABSTRACT: There is an increasing interest in developing learning analytic techniques for 
the analysis, and support of, high-quality learning discourse. This paper maps the terrain 
of discourse-centric learning analytics (DCLA), outlining the distinctive contribution of 
DCLA and outlining a definition for the field moving forwards. It is our claim that DCLA 
provides the opportunity to explore the ways in which discourse of various forms both 
resources and evidences learning; the ways in which small and large groups, and 
individuals, make and share meaning together through their language use; and the 
particular types of language — from discipline specific, to argumentative and socio- 
emotional — associated with positive learning outcomes. DCLA is thus not merely a 
computational aid to help detect or evidence "good" and "bad" performance (the focus 
of many kinds of analytics), but a tool to help investigate questions of interest to 
researchers, practitioners, and ultimately learners. The paper ends with three core 
issues for DCLA researchers — the challenge of context in relation to DCLA; the various 
systems required for DCLA to be effective; and the means through which DCLA might be 
delivered for maximum impact at the micro (e.g., learner), meso (e.g., school), and 
macro (e.g., government) levels. 
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1 INTRODUCTION 

"Learning Analytics is an emerging research field and design discipline that occupies the 'middle space' 
between the learning sciences/educational research and the use of computational techniques to 
capture and analyze data (Suthers & Verbert, 2013)" (Knight, Buckingham Shum, & Littleton, 2014, p. 1). 
One interest for learning analytics is in its potential for the analysis of learning processes, and discourse 
data (Buckingham Shum & Ferguson, 2012). In 2014 we saw the fourth Learning Analytics and 
Knowledge (LAK) conference and the second Discourse-Centric Learning Analytics (DCLA) workshop 
(Buckingham Shum, de Laat, De Liddo, Ferguson, & Whitelock, 2014) as a part of that conference. It now 
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seems a good time to reflect on this developing field. In this paper, we discuss this sub-area of learning 
analytics — discourse-centric learning analytics — which we refer to specifically in order to foreground 
our interest in the middle space between learning and analytics (Suthers, Lund, Rose, Teplovs, & Law, 
2013; Suthers & Verbert, 2013); an interest not only in computational-analytic techniques for discourse 
(the “DC" of DCLA) but in the explicit learning implications of those techniques, which should be 
grounded in our understanding of educationally salient discourse. This paper maps the terrain of the 
work falling under DCLA, describing the relationship between theory and methods, the various targets of 
analysis, and the kinds of learning claims at which those analyses might be targeted. We do not intend 
to put forward any particular stance with regard to the relationship between discourse and learning, but 
rather by mapping the terrain, we highlight necessary considerations in any well-motivated approach to 
DCLA as a distinctive research area. 


If learning analytics hopes to build on and contribute to learning theory, researchers should consider the 
ways in which learning analytics — in this case discourse-centric learning analytics — relates to, and is 
different from, other research on related topics. What is it that makes something DCLA, rather than 
natural-language processing, text-based machine learning, or some other form of learning analytics? The 
interest in developing DCLA techniques is in part driven by recognition of its potential bi-directional 
contribution; as we discuss further below, DCLA might take two forms: bottom up (inductive, 
exploratory), or top down (deductive, confirmatory). In either case, we must consider how the general 
approach relates to a distinctive sub-discipline of DCLA, and its relationship to existing theory. While in 
this paper we are specifically interested in DCLA, we anticipate that similar issues arise for other sub- 
domains of learning analytics. We consider these challenges in the context of DCLA, concluding with 
three key questions relating to context, systems, and feedback mechanisms. The paper thus provides a 
mapping of the DCLA terrain and its distinctive contribution, providing a focal point for DCLA's key 
challenges. A key component of our claim is that DCLA provides the opportunity to explore the ways in 
which discourse of various forms supports and evidences learning; the ways in which small and large 
groups, and individuals make and share meaning together through their language use; and the particular 
types of language — from discipline specific, to argumentative and socio-emotional — associated with 
positive learning outcomes. DCLA is thus not merely a computational aid to help detect or evidence 
"good" and "bad" performance (the focus of many kinds of analytics), but a tool to help investigate 
questions regarding the nature of discourse as learning, which are of interest to researchers, 
practitioners, and ultimately learners. 

1.1 Preliminary Definitions for DCLA 

Given its potential to support and investigate discourse in learning contexts, DCLA is thus of 
considerable interest. Whether in the context of Massive Open Online Courses (MOOCs), traditional 
university tuition, or more informal writing on the web, the automated or semi-automated analysis of 
linguistic data is increasingly prominent (see for example the classified sources in Table 1). Yet, it might 
well be noted that such analysis is not particularly new. Analysis of discourse and its communicative or 
illocutionary force is well established, and the ways in which new technologies mediate or alter such 
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properties are at least as old as Socrates' complaint regarding the diminished value of written argument 
(Plato, 1999). Given this context, it is important to consider the ways in which DCLA might differ in focus 
from related fields, what its potential is, what perspectives are brought to bear on the consideration of 
DCLA as opposed to other domains, and what are the defining features of DCLA. 


The trivial response, of course, is that DCLA is just the application of learning sciences, and analytic 
approaches, to student discourse data. However, insofar as this leaves unspecified definitional questions 
regarding "discourse," "learning analytics," and indeed "centric," this is clearly inadequate. To give an 
example, imagine a case in which a learner's motivation is assessed during the writing of a formal 
assessment (via non-linguistic proxies), and this motivation is fed back to him or her in some meaningful 
way. While certainly we might say this is learning analytics, and indeed we could accept that there is 
"discourse" data involved (the written assessment), it seems less clear to us that we would wish to 
describe such analytics as Discours e-Centric Learning Analytics. This paper describes our developing 
perspective on this issue, and marks out the ground within which we think DCLA lies. In doing so, we 
describe a set of illustrative examples drawn from our own work at the Open University. These are not 
intended to be exhaustive, nor indeed exemplary, but simply exemplifications of the types of space 
within which we conceive DCLA, and the pressure points for such work. 


A key focus of some earlier work (see for example, Scheuer, Loll, Pinkwart, and McLaren's 2010 review 
of the state of the art in computer-supported argumentation), has been the potential of automated 
analysis for discourse data — a key pivot point for DCLA. That review, and an earlier review (Clark, 
Sampson, Weinberger, & Erkens, 2007) of argumentation in Computer-Supported Collaborative Learning 
(CSCL) environments, largely focus on existing argumentation frameworks — that is, they take the "top- 
down" theory-driven approach. However, it is important to note the potential of DCLA approaches for 
contributing back to theory, in — well conceptualized — bottom-up, data-driven approaches, as we 
discuss in the section "Top-down and Bottom-up Approaches" below. 


The first DCLA workshop (Buckingham Shum et al., 2013) proposed a mission statement for DCLA, to 
"Devise and validate analytics that look beyond surface measures in order to quantify linguistic proxies 
for deeper learning." Yet, we hold the view that, in a very real sense, linguistic activity is not simply an 
indicator (or proxy) for deeper learning, it is often the site of that learning. That is, our learning is not 
just displayed through discourse; the discourse forms a fundamental site of that learning. Indeed, Table 
1 below gives an indication of some focal points for DCLA, which apply across the varieties of data that 
might properly be considered "discourse" (text and spoken-chat data, longer written monologues, CSCL 
interactions and so on). In each case, one can imagine — as is indicated in the columns — clear 
examples of educational foci, and benefits to such analysis. We highlight though, that as one moves 
down the rows, there is a shift in focus from individuals, to two types of small-group processes (Stahl, 
2010); in the first two — subject knowledge and rhetorical capabilities and tendencies — the emphasis is 
on individual capabilities and knowledge; in the third, it is on the ways in which people co-operate and 
share knowledge (the collective level); while in the fourth, discourse is taken to be not only indicative of 
learning, but constitutive of that learning in a collaborate, co-constructive sense. 
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Table 1. Discourse Functions, Focus, Content, and Example DCLA techniques* 


Function (from 
Knight, 2013) 

Focus 

Content 

Example DCLA techniques 

Supporting 

individuals' 

subject learning 

Advancement 

of subject 
knowledge. 

Propositional, semantic 

content of what is said. 

Appropriate reference to 
entities and relationships 

from the curriculum. 

Techniques for recognizing named 
entities, esp. ontology-driven 
techniques for domain specific 
entity-relationship extraction (e.g., 
history; biology) (Rose et al., 2008; 
Tablan, Roberts, Cunningham, & 
Bontcheva, 2013). 

Supporting 
psychological 
development — 

the 

development of 

social and 

reasoning skills 

Development 

of 

argumentation 

skills or 

dispositions. 

The rhetorical content of 

language. How language is 
used to position and make 
claims/discourse moves in 
discourse or writing. 

Techniques for extracting 
rhetorical and argumentation 
forms (Clark et al., 2007; De Liddo, 
Sandor, & Buckingham Shum, 

2012; Scheuer et al., 2010; 
Weinberger & Fischer, 2006). 

Promoting 

group 

understanding 
or commonality 

The 

interactional 

nature of 

collaborative 

discourse, 

networks of 

documents. 

Who says what to whom? 

The actors constituting the 
interpersonal context. 

Techniques for identifying 
interlocutors in dialogue, and more 

diffuse social networks of 

strong/weak ties via different 
channels (Haythornthwaite & de 

Laat, 2010; Oshima, Oshima, & 
Matsuzawa, 2012; Sie et al., 2012). 

Enabling 
sharing of ideas 

that can be 

improved 

together 

Resources that 

emerge 

through the 
discourse, as 
"improvable 
objects" 

(Wells, 1999). 

The ways discourse — 
particularly in CSCL contexts 

— resources future 

discourse and artefact 

development. The temporal 
and contextual (including 
social) development of 

ideas. 

The temporal and interactional 
nature of the above 3 rows, and 

the nature of the shifts between 

states — the pragmatic context 
(Bannert, Reimann, & Sonnenberg, 
2013; Furberg & Ludvigsen, 2008; 

Lid & Suthers, 2003; Medina & 
Suthers, 2009; Rosen, Miagkikh, & 
Suthers, 2011; Suthers & Desiato, 
2012; Suthers & Medina, 2010). 


*Table extended from Knight and Littleton (2013). 


The purpose of the table is to highlight the multiple ways in which discourse may be probed, the 
different properties of such analysis, and the ways in which such analysis might interact. This is 
particularly important if, through our data collection, task setting, and analytic methods, we reify a 
particular perspective on the purposes of discourse that neglects the multi-functionality of discourse. 
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This is true in both the context of summative and formative assessments of discourse quality; we should 
be careful to avoid giving feedback on only one aspect of discourse, of obscuring the nature of the 
discourse in attempts to reduce data to manageable, neat categories (enabling, for example, 
visualization and dashboards), and of failing to appropriately scaffold both teachers and students in their 
use of talk for learning. In the latter case, alongside those concerns, the issue of performativity (see, for 
example, Ball, 1999) is raised — wherein those aspects of a curriculum directly assessed, become those 
aspects most focused on in classroom contexts. Indeed (as with any assessment instrument) there is the 
risk that as certain forms of DCLA become widespread, teaching and assessment become driven by 
those facets of discourse made visible by these techniques; that we value what we can measure, rather 
than measuring what we value (Wells & Claxton, 2002). Moreover, while quantification offers analytic 
opportunity, it is not the end goal of learning analytics properly understood; closing the loop for 
meaningful feedback is crucial to differentiating learning analytics from other approaches. This was the 
focus of the second workshop (Buckingham Shum et al., 2014), asking "Once researchers have developed 
and validated discourse-centric analytics, how can these be successfully deployed at scale to support 
learning?" Understanding this full cycle is crucial, and it is to this issue that we turn in the section "The 
Learning Analytics Lifecycle." While DCLA of various sorts may be productive, we should be aware of the 
interpretive flexibility of the tools we develop, and the pedagogic, assessment, and epistemological 
context of which they form a part (Knight et al., 2014). However, if we are able to operationalize some 
stable and general categories and patterns in discourse use, then we can model such patterns in 
classifiers, recognizing that such modelling involves a balance around acceptable levels of information 
loss (Rose & Tovares, 2015). 


In the following section ("The Learning Analytics Lifecycle") we discuss how this process might occur, 
moving on to discuss the potential for "Top-down and Bottom-up Approaches" in this analytics lifecycle. 
We then move on to "DCLA: UNDERSTANDING WHERE DISCOURSE MEETS ANALYTICS," discussing some 
specific examples of Discourse Sites and analytic levels, around which our concluding definition and 
questions are focused. 

1.2 The Learning Analytics Lifecycle 

By an "analytics lifecycle" we mean the cycle through which a need or desire for an analytic technique is 
identified, the analytic itself is developed, and the new analytic tool is then implemented in some form. 
Consider the developed learning analytics lifecycle (Clow, 2012) that moves through learners, data from 
or about learners, processing of data into metrics, and intervention. The process of "evidence centered 
design" (Mislevy, Behrens, Dicerbo, & Levy, 2012) follows a similar pattern, beginning 1) with the 
identification of target constructs, then 2) identifying behaviours indicative of those constructs, before 
3) developing tasks likely to elicit those salient indicators. It is apparent, then, that DCLA should consider 
the types of tasks in which learners — formally or informally — engage, and the types of data that may 
be captured about those contexts. Theorizing is needed to consider how to process this data into 
appropriate metrics — how to interpret the data. Once metrics are developed, consideration of 
interventions should focus on ways in which data might be meaningfully represented either for students 
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or for expert educators in support of learners (in line with Clow's point that interventions do not 
necessarily need to involve a return of data to students). 


Consideration of other broad models of analytic cycles including that of learning analytics (Elias, 2011) 
— as summarized in Table 2 — indicate a broad alignment, in particular with the need to define goals (in 
our context, learning goals), in making decisions regarding data selection and capture. Each of the 
example cycles noted in the columns of Table 2 involves a set of key steps, from data selection through 
to use. It is important to note that, even at the stages of selecting and processing data, a range of 
irreducible elements involving theoretical models, target constructs, and data constraints are involved. 
For example, the segmentation of discourse data into chunks for processing is both a pragmatic decision 
regarding the selection of statistical technique, and also a theoretical one around the units at which data 
may be meaningfully discussed. A crucial question to ask at each DCLA stage — from task design and 
data collection to the steps of analysis and feedback to students — is "what is it for?" In this sense, DCLA 
is positioned in the "middle space." 


Table 2. Comparison of Analytics frameworks and models (Elias, 2011, p. 10) 


Knowledge 

Continuum 

Five Steps of 
Analytics 

Web Analytics 
Objectives 

Collective 

Applications Model 

Processes of Learning 
Analytics 

Data 

Capture 

Define Goals 

Select 

Select 

Measure 

Capture 

Capture 

Information 

Report 

Aggregate 

Aggregate & Report 

Knowledge 

Predict 

Process 

Predict 

Wisdom 

Act 

Use 

Display 

Use 

Refine 



Refine 



Share 


Share 


Given, then, the emphasis of various analytic models on selecting data and defining goals, an interesting 
question is raised — to what extent can "bottom up" approaches play a role in developing our theory 
and empirical understanding of phenomena, given the need for theory in data selection? 

1.3 Top-down and Bottom-up Approaches 

As we note in the introduction, the interest in DCLA is in part driven by recognition of its potential bi¬ 
directional contribution; DCLA might take two forms: bottom up (inductive, exploratory), or top down 
(deductive, confirmatory). Either might appropriately address a learning analytic development cycle — 
feeding into how we understand our desired construct, the types of behaviours associated with it, or the 
kinds of feedback likely to support some particular facet of learning. 
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In the former case, discourse data and learning outcomes are mined for patterns indicative of particular 
associations between discourse use and learning. In the latter, prior educational research is applied 
using new analytic techniques. As Cooper (2012) notes, both types are important contributions to the 
field of analytics. As Gibson (2013) points out (Table 3), data-driven approaches are different; the 
potential of such models is to simulate sophisticated aspects of higher order learning (with 
commensurate ethical concerns about auto-classifying students (Johnson, 2013). In such a view, 
inductive ("data-driven") approaches are used to derive data, from which hypotheses can be produced 
(validated), while deductive approaches (the traditional scientific hypothetico-deductive model) 
construct hypotheses to test through data collection and analysis. 


Table 3. Comparison between traditional and data-driven science methods (Gibson, 2013, p. 6) 



Traditional scientific method 

Data-driven science method 

Step 1 

Ask a Question 

Ask a Question 

Step 2 

Do Background Research 

Do Background Research 

Step 3 

Construct a Hypothesis 

Simulate Theory or Search for Patterns 

Step 4 

Test Hypothesis with Experiments 

Generate and Analyze Data 

Step 5 

Analyze Data and Draw a Conclusion 

Conduct Validation Experiments 

Step 6 

Communicate Results 

Communicate Results 


In both cases, the relationship between prior theorizing and empirical work, and new techniques should 
be established. Validation is fundamental here, and understanding the ways in which concepts are 
operationalized across the various manual, semi-manual, and automated methods is important (see for 
example, Chi, 1997). Of course, there are also cases where modelling computationally can inform 
theory. For example, Reimann (2009) discusses process models that analyze sequences of events — of 
relevance here insofar as they may be identified automatically from log data — in contrast to more 
traditional approaches that seek to model the variance in the occurrence of dependent variables. In this 
discussion, Reimann notes that 

Process mining can serve a number of purposes, among them: (a) Discovery—No a priori 
model exists. Based on an event log, a model is constructed; (b) Conformance—An a priori 
model exists. Event logs are used to determine the extent to which the enacted 
collaboration corresponds to the model; (c) Extension—An a priori model exists. The goal 
is not to test but to extend the model, for instance with performance data (e.g., durations 
of activities). (Reimann, 2009, p. 251) 

As Reimann notes in later work, 
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For [Educational Data Mining] EDM to contribute more directly to theory building, it 
needs to be applied to data that measures theoretically relevant properties and 
mechanisms. These are not necessarily found in the log files of software that has been 
designed for practical educational rather than research purposes. It is unlikely that we will 
find many theoretically interesting relations by looking at more of the essentially same 
kind of data. "Big data" and "more data" are not identical with conceptually "rich data" 
and "deep data" that capture not only multilayered phenomena but also a rich account of 
learning contexts. The latter, we believe, could be a productive methodological direction. 
(Reimann, Markauskaite, & Bannert, 2014, p. 11) 


Crucially, the two aspects of analysis that Chi (1997, p. 311) notes in relation to quantitative and verbal 
analyses hold true here too: first, generation of the right questions is crucial for determining the types of 
analyses one conducts; and second, understanding the means (the "mechanics") through which one 
conducts that analysis is a crucial consideration, not to be removed from (nor integrated in) the first. 
Similarly, we should take lessons about the translation of research to practice, and the transfer of 
practice from site to site into account in our analytics, bearing in mind that "educational data for 
learning analytics is context specific and variables carry different meanings and can have different 
implications across educational institutions and area of studies" (Ifenthaler & Widanapathirana, 2014, p. 
1 ). 

2 DCLA: UNDERSTANDING WHERE DISCOURSE MEETS ANALYTICS 


The preceding discussion of learning analytics lifecycles, and the role of top-down and bottom-up 
processes indicates that, although there is clear interest in the application of existing educational 
research to online contexts, particularly where analysis and learning support may be automated, more 
research may be needed to understand these applications. That is, the application of our existing theory 
to the context of novel forms of data — the types of data capturable, the contexts in which the data is 
created and captured, and the differences in the medium of online contexts — may need further 
research. 


For example, we have a strong interest in "exploratory dialogue" (see, for example, Mercer & Littleton, 
2007), which in small group contexts has a demonstrated relationship with improved learning outcomes. 
Flowever, little research has been conducted on exploratory dialogue in online contexts (see, for 
example, Littleton & Whitelock, 2005), with only a few studies of such asynchronous dialogue (see, for 
example, Ferguson, Whitelock, & Littleton, 2010), and none that we are aware of in the context of large 
multi-party and multi-modal conference chat systems. Yet, it is clear that there are differences between 
face-to-face and online communication, and many online contexts provide different types of 
opportunities for communication. This is an important point regarding a "middle space" between the 
learning sciences and analytic techniques (Suthers & Verbert, 2013); even a meeting in the middle space 
can be approached from either side, and such trajectories affect the ways we action our theories. If our 
operationalization of prior (offline) educational research is driven by the analytic techniques available, 
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then the types of validation in which we must subsequently engage, and the ways in which we approach 
the on- and offline dialogue as related or not, should be considered. Similarly, our analytic techniques 
should not be limited by the ways in which educational dialogue has previously been operationalized; 
new analytic techniques afford new opportunities to theorize and reconsider what constitutes 
productive dialogue. 


Indeed, if we consider the classic "quantification of verbal analysis" text (Chi, 1997), in that (manual) 
case, the point is made that all stages of analysis involve both top-down and bottom-up processing; 
coding is driven by top-down theory, but the application of codes is refined through bottom-up 
protocols (Chi, 1997, pp. 309-310). To give a concrete example in the machine learning case, Rose and 
Tovares note, "more attention should be given to the problem of representing data appropriately" (Rose 
& Tovares, 2015, p. 6), particularly with regard to the issue of segmentation where different levels of 
granularity in segmentation for the training and classification of various forms of natural language data 
may have an impact. This concern is a complex one, and in earlier work Arguello and Rose (2006) 
propose that regarding topic segmentation, researchers first need a definition of "topic," which should 
be: 

1. Reproducible by human annotators 

2. Not reliant on domain-specific or task-dependent knowledge 

3. Grounded in generally accepted principles of discourse structure (in order that shifts in topic are 
recognizable from surface characteristics of the dialogue) 


Thus, although in that case hidden Markov models were used to detect topic shifts, the point is to 
reproduce a human-recognized topic model (rather than anything more ephemeral). That is, although 
"bottom-up" methods may create productive means of segmentation, the relationship of such methods 
— and their outputs — to theoretically grounded assumptions should not be forgotten. 

In the next two sub-sections, we give some examples of contexts in which discourse data is produced, 
and is associated with learning outcomes. In the section on discourse sites, we introduce discourse 
contexts in which data is collected, and the implications of those learning contexts and the data 
collected for the analytics we conduct. This section is broadly concerned with the "top down" approach 
to DCLA, derived from existing learning theory and empirical work. In the section on learning analytics, 
we discuss a broad set of analytic techniques for discourse data, and the implications of these for our 
learning-discourse analysis, and feedback representations. This section is broadly concerned with the 
"bottom-up" approach to DCLA, derived from rapidly advancing analytic techniques. 

2.1 Discourse Sites 


The aim of this (and the next) section is to introduce some examples of work on the "sites" of discourse 
noted in Table 1, around which there has been interest in developing analytics. We use the examples to 
illustrate the ways in which people develop their reasoning through dialogue, build arguments in CSCL 
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environments, and make rhetorical moves in written texts. Returning to Table 1, recall rows that focus 
on 

1. The kinds of subject knowledge people evidence 

2. The kinds of argument skills or dispositions they evidence 

3. The ways in which learners interact with each other and with ranges of documents 

4. The ways in which meaning is built up through shared interaction on documents 


While identifying subject knowledge (content, entities, concepts, etc.) is important, it is not the only 
crucial element. So too for interaction — whether with citations, connecting "nodes" in CSCL 
knowledge-building environments, or simply through response to others' arguments or moves (e.g., 
responding to a question). Instead, combinations of these important features of a text are crucial to 
understanding what is going on. The first two rows of the table, then, are about individual levels of 
knowledge, the third about collective levels, and the fourth about collaborative co-construction. 


We use the following sections to highlight some exemplar (but by no means exhaustive) learning 
sciences literature around discourse, taking as our focal point theorizing around argumentation and 
reasoning. In the following paragraphs, we present three types of discourse data: 1) unstructured 
dialogue, 2) free-text, and 3) dialogue within structured CSCL environments. In each case, we are 
interested in the forms of data available and the contexts in which they are obtained. These are of 
particular interest for the following section, which discusses the ways in which such data has been 
treated analytically, and raises some potential areas of interest. The point here is to note particularly 
how theory informs our collection of, and analysis of, data and to foreground the levels at which DCLA 
might operate. 


2.1.1 Unstructured dialogue 

Perhaps the most obvious form of discourse data is the kind of computer-mediated communication seen 
in "chat" programs, and free-form dialogue of classrooms (and indeed, everyday conversation). In 
everyday classroom contexts, teachers model the use, and pay attention to the uptake of, key content 
and rhetorical terms in student dialogue (rows 1 and 2 of the table). A common complaint in school 
classrooms at least, is that whole-class discussion focused on questioning tends to focus too much on 
these levels of — individual — content-based understanding, rather than whole class commonality 
(Alexander, 2008). For these levels of discourse use (the latter 2 rows of the table) classroom 
"assessment for learning" techniques such as "mini-whiteboards" and interactive whiteboards 
(Hennessy & Warwick, 2010) for all students to display an answer, or more dialogic modes of discussion 
(Alexander, 2008), may be more important. In our own work we have a strong interest in the ways in 
which unstructured dialogue is used to co-construct common knowledge (Edwards & Mercer, 1987; 
Littleton & Mercer, 2013), contributing to the strong consensus among researchers that in a variety of 
contexts, high-quality dialogue is associated with learning (see the collection edited by Littleton & Howe, 
2010). A core component of this research — along with that around Accountable Talk (Michaels, 
O'Connor, Hall, & Resnick, 2002; Resnick, 2001) — is the analysis of how ideas (including subject 
knowledge) are deployed in arguments, and taken up by others, to help build new knowledge together. 
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Fundamental to this process is critical but constructive engagement with others' ideas, participation 
from all members, respectful challenges and justifications, and the consideration of opinions before 
agreement is sought — in short, to make the reasoning visible in the dialogue (Mercer & Littleton, 2007). 


2.1.2 Written texts 

Of course, spoken dialogue through short exchanges is not the only site of learning. In particular in 
formal learning contexts — although by no means exclusively — much learning also occurs through the 
construction of, and is represented in, longer free-form written texts. Thus, one means through which 
the kind of co-construction noted in the 4 th row occurs, is in the interaction with "the given," with 
written texts of various forms (see, for example, Knight & Littleton, 2015). Indeed, this too should be a 
core area of interest for DCLA researchers — how learners engage with texts, to construct longer pieces 
of writing, and how that writing is a site for, and representation of, their learning. Again, subject 
knowledge is key here, but so too is genre — including the kinds of "rhetorical moves" (or arguments) 
available, and so on. As Simsek, Buckingham Shum, Sandor, De Liddo, and Ferguson (2013, p. 3) note, 
analysis of texts for factual statements provides only a partial picture; in scientific research publications, 
authors make connections between claims of various sorts, including supporting, refuting, and so on, 
and this meta-discourse is fundamental to analysis of such papers. 

Again, we see the importance not only of subject knowledge, but argumentation skill (rhetorical moves 
made), use of other's ideas (through citation and background knowledge), and genre-specific knowledge 
building (each discipline having its own "style" of making sometimes similar claims). Additional Natural 
Language Processing (NLP) approaches of interest to DCLA can be found in the emerging field of 
Argument Mining, which sits at the convergence of machine learning, information retrieval and 
argumentation theory (Palau & Moens, 2009). Again, the four rows of the table can be explicated in 
terms of this data — at the first two levels (or rows), the use of particular writing genres or scripts, and 
deployment of key concept terms, with the third and fourth rows seen in deeper engagement with 
genres, and deeper interactive reasoning around texts. 

2.1.3 Structured dialogue and CSCL environments 

As we note above, the relationship between "given," or sole-author student created texts and student 
dialogue are not firm. Indeed, there have been calls for "abandoning the forced dichotomy between two 
genres of collaboration tool" (Enyedy & Hoadley, 2006, p. 414), calling for fusion between information 
(document), and communication interfaces in part to facilitate the kinds of learning identified in the 
third and fourth rows of Table 1. The use of CSCL tools to support document-oriented, or less directed 
reasoning tasks is widespread; aiming to support the kinds of individual learning identified in the first 
two rows of Table 1. Much work has been conducted on developing environments that support, and 
make available for researchers, particular types of dialogue. Importantly though, while de Vries, Lund, 
and Baker (2002), to take one example, note important characteristics of productive epistemic dialogue, 
which bear striking resemblance to those described in this paper, they note that such dialogue is not 
"automatically" produced in structured environments. Indeed, Dillenbourg (2002) notes that some such 
environments risk "over structuring" and thus restricting the use of important types of dialogue. We 
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note, then, that while design may reduce computational difficulties (for example, by introducing 
threading to discussions), the points made in this paper with respect to the importance of context are 
still fundamental to understanding the dynamic features of dialogue through which learning is co¬ 
constructed. Computer environments may be seen as complementary to such dialogue, in particular 
where they formalize, through the user interface conceptual model, some of the systems through which 
exploratory and accountable dialogue are more likely to occur — the "ground rules" of each. It is thus 
that systems have been developed specifically to support particular types of formalized argumentation 
schema (Clark et al., 2007; De Liddo, Buckingham Shum, Quinto, Bachler, & Cannavacciuolo, 2011; 
Scheuer et al., 2010; Weinberger, Ertl, Fischer, & Mandl, 2005; Weinberger & Fischer, 2006). Again, we 
highlight the range of ways in which CSCL argumentation aligns with the table with some scripting aimed 
at individual learning gains (particularly around argumentation schema), and other methods aimed to 
aid in intersubjective meaning making. 


2.2 Analytic Levels 

We have highlighted some of the data forms and contexts of learning discourse above. The aim of this 
section, then, is to discuss the data-types and learning theory in light of analytic techniques. We note 
that even in contexts in which inductive or exploratory approaches to data analysis are used, the data 
itself is still imbued with theory insofar as the learning contexts from which that data is collected require 
theorizing; for example, genre detection requires a notion of "genre," and the collection of documents 
displaying such classes. Indeed, in the rather different context of transcribing audio and video 
recordings, Flammersley (2010) makes similar points: transcriptions cannot be said to be purely 
"constructed" as they bear an obvious and undisputable relationship to their object of representation, 
which is closely associated with their theoretical ontogeny, the prior knowledge of the transcriber, and 
the purposes for which the transcription is being created. Similarly though, they are not "given" direct 
insights into the world; shared understanding is constructed in the process of transcription itself and 
relies heavily on a (given) set of cultural and theoretical values including the best ways to represent 
given classes of behaviour (for example, whether to include notes on intonation). In this section, then, 
we highlight the ways in which theory should penetrate analytics for DCLA, and the complexity arising 
from the intersection between our data type (dialogue, text, CSCL) and analytic focus (subject 
knowledge, argumentation skill, interaction, and broader knowledge-building culture). 


2.2.1 The individual level 

If we consider again Table 1, recall that the first two rows regard individual level learning gains — in 
terms of content- or subject-based learning and personal psychological development. Analysis of such 
learning through discourse focuses on the ways in which individuals use terms, whether in individual or 
collective contexts; what is key is not the type of discourse data, but rather the analytic focus. The most 
basic kinds of analysis take simple "bag of word" or cue phrase approaches — in which classes of 
dialogue are assigned based on the presence of key words or phrases — to detection of concepts. Such 
approaches can be enriched with ontologies to match key phrases from some domain, with those 
present in the discourse data. More complex variants on this approach might include a broader feature 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


196 


JOURNAL OF LEARNING ANALYTICS 

(2015). Discourse-Centric Learning Analytics: Mapping the Terrain. Journal of Learning Analytics, 2(1), 185-209. 


SSLAR 


set — such as grammatical markers, particularly in the case of argumentation features — to analyze the 
presence of particular types of discourse. As such, analysis focuses not on the interactional features of 
discourse, which such techniques are not methodologically equipped to analyze, but on the individual's 
use of terms and styles of discourse, and their relationship to individual learning (of concepts, and 
perhaps of rhetorical moves of some kind, and so on). 


This kind of approach can be applied across dialogue, written text, and CSCL environment discourse 
data, although the former more naturally lends itself to these techniques. Written essays also offer 
opportunity for a variety of other semantic-related analyses, such as latent semantic analysis (LSA), 
which goes beyond cue-phrase analysis to look at underlying (latent) semantic content in the text. Such 
analyses can also be conducted on appropriately segmented dialogue data; for example, Epistemic 
Network Analysis (Shaffer et al., 2009) conducts a principal component analysis to explore connections 
between themes throughout a dialogue. A major advantage of CSCL environments for such analysis is 
the ability to build in "tagging" systems to enable learners to "tag" their entries with semantic labels, 
which can then be analyzed. In each of these cases, feedback can focus on the types of concepts and 
facts included in the discourse, and their incidence. Similarly, analysis for argumentation, rhetorical 
moves, and tendency (or disposition) to use such devices can also be analyzed through these methods. 
Again, feedback can focus on incidence of particular markers, but it can also — when combined with 
content analysis — focus on the ways concepts and facts are connected using argument-markers (for 
example, "it is argued that Broca's area is related to language because evidence from studies (refl, ref2, 
etc.)...," where Broca's and the references noted are conceptual or factual claims, and the marker 
"because" links them). Again, CSCL knowledge-building/argument-mapping environments are 
particularly well equipped for this kind of analysis, in which the "connections" learners make between 
claims (often represented as "nodes") are often pre-labelled, and thus in machine-readable format. 

These notions then cover the ways in which individuals use language to share meaning, and express 
their knowledge. However, we are also interested in how learners interact with each other using 
dialogue, how they move beyond expressing to "taking up." There are two levels of interest here. The 
first (and third row of the table) regards the ways in which learners interact with each other and ranges 
of documents, and how they pass information to each other, which is then used in the collective 
context. The second regards how meaning is built up through shared interaction, how knowledge is co¬ 
constructed. Again, the first two rows of the table are about individual levels of knowledge, the third 
about collective levels, and the fourth about collaborative co-construction. 


2.2.2 The collective small group level 

At the collective level of analysis, the focus of attention shifts from expressions by individuals, to the 
ways in which individuals — and the things that they say — interact. Thus, for example, Rose and 
colleagues have explored transactivity (sometimes called intertextuality) in small groups (Sionti, Ai, 
Rose, & Resnick, 2011; Stahl & Rose, 2011), dyads (Gweon, Jun, Lee, Finger, & Rose, 2011), whole-class 
discussions (Ai, Sionti, Wang, & Rose, 2010), and its use for summarizing group discussions (Joshi & 
Rose, 2007). Similarly, the ways in which written texts can be used to resource further written texts — in 
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or out of relatively structured environments — is clearly important (Enyedy & Hoadley, 2006; Knight & 
Littleton, 2015; Lid & Suthers, 2003; Suthers, Vatrapu, Medina, & Dwyer, 2007). Again, the structured 
nature of CSCL environments supports analysis to explore the ways in which ideas are "taken up" by 
others, and knowledge is pooled — collectively — to create (perhaps individual) answers. For example, 
"contingency graphs" in CSCL environments (Medina & Suthers, 2009) provide a visual insight into the 
ways in which one "idea" (a node), is "contingent upon" earlier ones. In each of these cases, analysis can 
focus on how terms are stated by an individual or given text, and then subsequently repeated by other 
individuals. More sophisticated analysis — particularly in the case of longer written texts — might also 
focus on the latent-semantic content of a student-produced text in comparison to those from which it is 
supposed to draw (see for example, Hastings, Hughes, Magliano, Goldman, & Lawless, 2012). Indeed, 
experimental work can also be modelled on Azmitia and Montomery (1993), who artificially 
manipulated which pieces of information participants in a group had, such that information needed to 
be passed between members in order to complete a task. 


However, as Stahl (2013) recently noted, in such cases transactivity is being explored at the analytic level 
of individuals who have (differing) partial information regarding a problem that requires pooled 
information to solve it, rather than at the group level, which has come to be of great interest to the CSCL 
community. This focus on individuals in collaborative contexts, in contrast to collaborative units, is 
common to much group-work research; for example, in exploring transactivity, Azmitia and 
Montgomery's (1993) interest was in dialogue used to build upon a partner's explicit reasoning 
statements, rather than on the language used to co-construct knowledge. This, then, brings us to our 
last interest — that of the co-constructive level (the 4 th row) of group-cognition. 


2.2.3 The co-constructive small-group level 

A fourth level of analysis is at the group-co-constructive level. Indeed, notions of transactivity are highly 
relevant here; for example, Sionti et al. (2011) summarizes notions of transactivity as sharing a 
requirement for interlocutors' reasoning to be explicitly displayed, and usually for contributions 
between speakers to be connected such that a speaker's utterances resource future utterances. In 
addition to this interesting construct, it is also worth mentioning other work — relating more closely to 
active participation and the reasoned consideration of others' arguments — in which the notion of 
"heteroglossia" is explored. Heteroglossia (Bakhtin, 1986) is related to the multi-vocality of perspective, 
the characteristic of a text as displaying, and being open to, multiple views — a significant element of 
dialogic education (Wegerif, 2011). Building on Martin and White's (2005) description of dialogic 
expansion (in which alternative positions are available), and dialogic contraction (in which the scope of 
permitted perspectives is restricted), heteroglossia has recently been operationalized in the 
computational linguistics context as: 

the extent to which a speaker shows openness to the existence of other perspectives 
apart from the one that is reflected in the propositional content of the assertion being 
made... Within our Heteroglossia analysis, assertions framed in such a way as to 
acknowledge that others may or may not agree, are identified as heteroglossic. We 
describe it as identifying wording choices that do or don't treat other perspectives than 
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what is expressed in the propositional content of the assertion as open for consideration 
within the continuing discussion. (Rose & Tovares, 2015, pp. 10-11) 


In the context of a science classroom task, this has been operationalized at the coding level for a four- 
part scheme that takes co-constructive dialogue as an "openness to the other" in the way phrases are 
used. By building in further features at the small-group level, such as looking at dialogue-acts for 
interaction properties within the group (Erkens & Janssen, 2008; Krai & Cerisara, 2012; Stolcke et al., 
2000), we can draw further conclusions about how small-groups work together in co-constructive ways 
to create new meaning. In such analytic approaches, the focus is on the ways in which terms are "taken 
up" by others, and the collective accretion of knowledge over time, as indicated in the fourth row of the 
table. To analyze such learning, a temporal analysis of features — understanding how terms are used, by 
whom, and how they relate to each other — is crucial (Mercer, 2008). While temporality is, of course, 
relevant to other areas of learning (including rows 1-3), it is particularly salient to this fourth, co- 
constructive level. 


2.3 Section Summary 

In learning contexts, we are interested not only in the deployment of particular terms, but of their 
effective deployment in contexts that indicate an understanding of the inferential properties of the term 
to other concepts. Analytics that explore key words in abstracted ways may obscure the misuse of 
terms, or — moreover — their simple copying from texts that students use for contextualizing purposes, 
such as task instructions. The pragmatic level of description is therefore important: syntactic or semantic 
levels of description can be blind to understanding what is being done in interaction. For example, the 
"same" question (in syntactic and semantic terms) might be asked at the beginning and end of a lesson, 
while serving different (pragmatic) functions: in the first instance, to gauge baseline understanding and 
provide a reference point for the second posing of the question, which is to see how the question may 
now be reinterpreted. Describing what is happening is not the same as understanding what is being 
done. 


We are mindful that, regarding the complexity of discourse data, one solution — for example through 
digital games (Gee, 2008; Shaffer, 2008) — is to "design in" for the types of language we want to teach, 
and analyze; providing opportunities for their display and capture in ways easy to process by machines. 
Certainly this point is important, and DCLA should consider the ways it might be informed by, and 
inform, such structured environments. The potential to build complex, discursive learning contexts is 
important and the full breadth of ways in which discourse is constitutive of learning should not be 
glossed by limiting analytic techniques. 

3 CONCLUSIONS: WHAT IS DCLA? 

There is a growing interest in, and availability of, data and techniques to analyze that data. Discourse is 
an important feature of learning, and fundamentally associated with the context of social learning 
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analytics, dispositional analytics, and other emerging analytic-sub-fields. We should strive to understand 
discourse in this wider context. A further driver here is the fact that interest in online text 
summarization tools, and support for writing both in and out of formal educational contexts will 
continue to grow — again, DCLA may play an important role here, perhaps complemented by other 
forms of analytic tools (such as social network analytics). 


3.1 A Definition for DCLA 


This paper has highlighted the ways in which we see DCLA as distinctive; in doing so, we have tried to 
contextualize it with respect to the wider research. We began the paper by noting two early focal points 
for DCLA: 

1. Devise and validate analytics that look beyond surface measures in order to quantify linguistic 
proxies for deeper learning 

2. Once researchers have developed and validated discourse-centric analytics, how can these be 
successfully deployed at scale to support learning? 

As we have highlighted throughout this paper, we must be cautious here — linguistic activity is not 
simply an indicator for deeper learning, it is the very site of that learning; when we talk to each other, 
write texts, engage with knowledge-building environments, we are not simply representing what is 
known — providing indicators for that knowledge — we are actively engaging in a co-constructive 
activity, which constitutes learning. With this understanding of high-quality learning discourse, we 
propose a definition of DCLA: 

DCLA focuses on analytics to support high-quality discourse for learning contexts; it 
consists of the analysis of discourse data, the creation of effective feedback to learners 
and educators, and the validation and theorizing of our analytic techniques. 

This definition, of course, leaves open the particular theorized account of discourse, and pedagogic 
approach associated with particular types of discourse. Throughout the paper, we have tried to highlight 
some core pivot points for DCLA, seeking to elaborate our understanding of learning analytics as centred 
around discourse data. In particular, we have noted the ways in which various sites of discourse might 
be seen as related to, or constitutive of, learning at a variety of analytic levels — from the individual to 
the small group. The definition given does not preclude the use of analytic devices for analysis of 
learning indicators in discourse, of interest particularly to evidence learning at an individual level, but 
rather it leaves open the space to explore the full range of levels at which learning and discourse are 
related. It also foregrounds the need for analytics centred on discourse to consider validation in 
transferring analysis from existing learning theory to analytic techniques, and vice-versa, as well as the 
need to consider the wider learning and feedback systems within which analytic techniques are 
deployed — particularly as targeted at learning outcomes. We close the paper with three key issues that 
researchers in DCLA will need to consider further: 1) the role of context, 2) DCLA systems, and 3) 
effective impactful feedback. 
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3.1.1 What is the relationship between discourse in different contexts (particularly on- and offline), 
and what are the implications for detection of particular types of discourse? 

As we note above, the differences between on- and offline instantiations of particular discourse-genres 
are complex, and this area will be of much importance. In order to apply "offline" research in online 
contexts, we must be sensitive to the differences and the possible need to "validate" such a shift. 
Similarly, lessons from online contexts may not translate directly to offline ones — not least because the 
types of data we can gather are rather different, and thus our insights may be different as well. For 
example, creating written texts using technologies — including collaboration tools such as etherpads or 
google docs — provides different affordances, and research data, to "pen-and-paper" exercises. 
Consider also the wealth of research on computer-mediated communication (CMC) that recognizes and 
explores differences in CMC and "offline" talk. DCLA researchers should consider the role of existing 
theory, and the place of analytics in that context. This issue thus concerns: 

1. The practical-context sensitivity of DCLA, and indeed all discourse analysis, regarding pedagogic, 
assessment, cultural, and individual differences aspects 

2. The theoretical-context sensitivity of DCLA and how alignments and mismatches between on- and 
offline contexts might be addressed 

3. The role of inductive and deductive techniques in advancing analytics and theory 

4. The need for validating learning analytics techniques 


3.1.2 What systems need to be in place for the successful deployment of DCLA? 

DCLA rely on particular tools for data capture and analysis. However, their use may also make 
commitments to particular pedagogic, assessment, and epistemological assumptions (Knight et al., 
2014). While tools have particular affordances (for example, twitter replies, forum threading, 
knowledge-mapping connections), these tools can be used in many learning contexts, and their data 
analyzed for many purposes. This highlights that the issue of "systems" relates not only to the analytic 
device, but also to the importance of meaningful activities in learning contexts for learning analytics. We 
agree that more attention should be paid to such claims, and that design strategies have strong 
potential for the development of meaningful, and enjoyable, learning activities centred on language use 
(Gee, 2008; Shaffer, 2008). Indeed, much work has been conducted on developing environments that 
support, and make available for researchers, particular types of discourse. DCLA can play a role in this. In 
considering the necessary systems for the successful deployment of DCLA we should thus consider the 
following: 

1. The technical factors involved in DCLA, the types of data we can (and want to) collect, and how they 
are processed 

2. The pedagogic and assessment context within which data is collected — collection of data should 
occur in purposeful, pedagogically valuable contexts 

3. An understanding of the contexts within which certain discourse types are likely to occur, such that 
we do not deploy the "wrong" analytic device in contexts where we would not expect its target class 
to occur 
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3.1.3 What is the most effective way to feedback results to learners, educators, and policy makers for 
maximum impact? 

As discussed above, the learning analytics cycle is not solely about the creation of numerical insights into 
the target-data discourse in DCLA. Rather, we are interested in how we can develop tools to support 
education. As such, understanding how best to feedback discourse supporting data — to whom, in what 
form, and from what data — is crucial. While computationally complex analytics might hold potential, 
simpler techniques, might also be impactful. For example, analytics that indicate which topics are being 
talked about, by whom, giving information such as turn taking — without evaluating "good" or "bad" 
performance — might provide pedagogic support, while being very computationally simple. In addition, 
learners, educators, and policy makers each have different needs; analytics may provide 
representational tools to resource discussion around learning impact at all levels. A part of analytics is 
creating representational tools to understand data we already have; this question relates to that data, 
and new data emerging from learning analytic techniques. Whether computationally simple or complex 
tools are deployed, interpretive flexibility plays a role. This issue thus concerns the following: 

1. The ways in which the same data can be collected, analyzed, and fed back to relevant stakeholders 
at the macro, meso, and micro levels (see Figure 1 below) 

2. Research around visualization, at the various levels, to support "in the moment" and "after the fact" 
discourse 

3. In addition to supporting learning through DCLA, research should also be conducted on how to 
support learning about DCLA — how learning situations can make best use of DCLA and in particular 
how at each level (macro-meso-micro) stakeholders can communicate with others around DCLA 
sensemaking (Knight, Buckingham Shum, & Littleton, 2013) 


Macro: 

region/state/national/international 


Meso: 

institution-wide 


r 


Micro: 

individual user actions 

(and hence cohort) 


Aggregation of user traces 
enriches meso + macro analytics 
with finer-grained process data 


Breadth * depth from macro 
+ meso levels add power to 
micro analytics 


Figure 1. The macro-meso-micro levels of analysis for 
learning analytics (Buckingham Shum, 2012, p. 3) 


We can envisage scenarios, therefore, in which leaders at the institutional level and above see 
aggregated DCLA (for example, as proxies for critical thinking, constructive knowledge building, or 
successful deployment of online forums), and are able to compare datasets from many contexts. The 
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extent to which comparisons can be made with integrity within and between contexts is, of course, an 
issue afflicting all types of analytics (educational or otherwise), such is the ease with which digital data 
may be fused. The particular roles that discourse plays in learning, which we have sought to illuminate in 
this paper, alert us to the risk of decontextualizing and comparing the different types of DCLA without 
due care. That being said, there is an intriguing array of opportunities within the middle space of 
learning sciences and computational techniques around discourse data; by staking the distinctive 
contribution of DCLA, we highlight this potential and the core challenges for DCLA researchers moving 
forwards. 
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